2025-04-02 00:12:52,729 [ 644316 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2025-04-02 00:12:52,730 [ 644316 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:97, check_args_and_update_paths) 2025-04-02 00:12:52,730 [ 644316 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:108, check_args_and_update_paths) 2025-04-02 00:12:52,730 [ 644316 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:110, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_bk5u9t --privileged --dns-search='.' --memory=30709035008 --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=6712d5cc610d -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS=" -rfEps --run-id=2 --color=no --durations=0 test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference -vvv" altinityinfra/integration-tests-runner:cd6390247eca '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: random-0.2, timeout-2.2.0, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False collecting ... collected 2 items test_s3_cluster/test.py::test_distributed_insert_select_with_replicated FAILED [ 50%] test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference FAILED [100%] =================================== FAILURES =================================== ________________ test_distributed_insert_select_with_replicated ________________ started_cluster = def test_distributed_insert_select_with_replicated(started_cluster): first_replica_first_shard = started_cluster.instances["s0_0_0"] second_replica_first_shard = started_cluster.instances["s0_0_1"] first_replica_first_shard.query( """DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC;""" ) first_replica_first_shard.query( """ CREATE TABLE insert_select_replicated_local ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/insert_select_with_replicated', '{replica}') ORDER BY (a, b); """ ) for replica in [first_replica_first_shard, second_replica_first_shard]: replica.query( """ SYSTEM STOP FETCHES; """ ) replica.query( """ SYSTEM STOP MERGES; """ ) > first_replica_first_shard.query( """ INSERT INTO insert_select_replicated_local SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV','a String, b UInt64' ) SETTINGS parallel_distributed_insert_select=1; """ ) test_s3_cluster/test.py:393: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 49, stderr: Received exception from server (version 24.12.2): E Code: 49. DB::Exception: Received from 172.16.1.10:9000. DB::Exception: Replica info is not initialized: While executing Remote. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d78d79b E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000867958c E 2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000008686b0b E 3. DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x00000000112e0637 E 4. DB::RemoteQueryExecutor::readAsync() @ 0x00000000112e1b76 E 5. DB::RemoteSource::tryGenerate() @ 0x0000000013aefa30 E 6. DB::ISource::work() @ 0x0000000013787127 E 7. DB::ExecutionThreadContext::executeTask() @ 0x00000000137a1947 E 8. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x0000000013794c25 E 9. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000013793fb6 E 10. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000137930f7 E 11. ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x000000000d86a102 E 12. void* std::__thread_proxy[abi:v15007]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000000d8714ba E 13. ? @ 0x00007fea98874ac3 E 14. ? @ 0x00007fea98906850 E . (LOGICAL_ERROR) E (query: INSERT INTO insert_select_replicated_local SELECT * FROM s3Cluster( E 'first_shard', E 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV','a String, b UInt64' E ) SETTINGS parallel_distributed_insert_select=1;) helpers/client.py:248: QueryRuntimeException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml data/clickhouse/part1.csv data/clickhouse/part123.csv data/database/part2.csv data/database/partition675.csv data/generated/file_0.csv data/generated/file_1.csv data/generated/file_10.csv data/generated/file_11.csv data/generated/file_12.csv data/generated/file_13.csv data/generated/file_14.csv data/generated/file_15.csv data/generated/file_16.csv data/generated/file_17.csv data/generated/file_18.csv data/generated/file_19.csv data/generated/file_2.csv data/generated/file_20.csv data/generated/file_21.csv data/generated/file_22.csv data/generated/file_23.csv data/generated/file_24.csv data/generated/file_25.csv data/generated/file_26.csv data/generated/file_27.csv data/generated/file_28.csv data/generated/file_29.csv data/generated/file_3.csv data/generated/file_30.csv data/generated/file_31.csv data/generated/file_32.csv data/generated/file_33.csv data/generated/file_34.csv data/generated/file_35.csv data/generated/file_36.csv data/generated/file_37.csv data/generated/file_38.csv data/generated/file_39.csv data/generated/file_4.csv data/generated/file_40.csv data/generated/file_41.csv data/generated/file_42.csv data/generated/file_43.csv data/generated/file_44.csv data/generated/file_45.csv data/generated/file_46.csv data/generated/file_47.csv data/generated/file_48.csv data/generated/file_49.csv data/generated/file_5.csv data/generated/file_50.csv data/generated/file_51.csv data/generated/file_52.csv data/generated/file_53.csv data/generated/file_54.csv data/generated/file_55.csv data/generated/file_56.csv data/generated/file_57.csv data/generated/file_58.csv data/generated/file_59.csv data/generated/file_6.csv data/generated/file_60.csv data/generated/file_61.csv data/generated/file_62.csv data/generated/file_63.csv data/generated/file_64.csv data/generated/file_65.csv data/generated/file_66.csv data/generated/file_67.csv data/generated/file_68.csv data/generated/file_69.csv data/generated/file_7.csv data/generated/file_70.csv data/generated/file_71.csv data/generated/file_72.csv data/generated/file_73.csv data/generated/file_74.csv data/generated/file_75.csv data/generated/file_76.csv data/generated/file_77.csv data/generated/file_78.csv data/generated/file_79.csv data/generated/file_8.csv data/generated/file_80.csv data/generated/file_81.csv data/generated/file_82.csv data/generated/file_83.csv data/generated/file_84.csv data/generated/file_85.csv data/generated/file_86.csv data/generated/file_87.csv data/generated/file_88.csv data/generated/file_89.csv data/generated/file_9.csv data/generated/file_90.csv data/generated/file_91.csv data/generated/file_92.csv data/generated/file_93.csv data/generated/file_94.csv data/generated/file_95.csv data/generated/file_96.csv data/generated/file_97.csv data/generated/file_98.csv data/generated/file_99.csv ---------------------------- Captured stderr setup ----------------------------- Command:[docker ps | wc -l] Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Stdout:net.ipv4.ip_local_port_range = 55000 65535 ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 ENV HOSTNAME 34bca11fe2c4 ENV SHLVL 0 ENV HOME /root ENV OLDPWD / ENV DOCKER_HELPER_TAG 5dc43a6382f0 ENV PYTHONUNBUFFERED 1 ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e ENV UBSAN_OPTIONS print_stacktrace=1 ENV PYTEST_ADDOPTS -rfEps --run-id=2 --color=no --durations=0 test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference -vvv ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge ENV COMPOSE_HTTP_TIMEOUT 600 ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ENV DOCKER_KERBERIZED_HADOOP_TAG latest ENV DOCKER_CHANNEL stable ENV DOCKER_CLIENT_TIMEOUT 300 ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e ENV PWD /ClickHouse/tests/integration ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config ENV TZ Etc/UTC ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java ENV DOCKER_BASE_TAG 6712d5cc610d ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 ENV LC_CTYPE C.UTF-8 ENV INTEGRATION_TESTS_RUN_ID 2 ENV WORKER_FREE_PORTS 30000 30001 30002 30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014 30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026 30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038 30039 30040 30041 30042 30043 30044 30045 30046 30047 30048 30049 ENV PYTEST_CURRENT_TEST test_s3_cluster/test.py::test_distributed_insert_select_with_replicated (setup) CLUSTER INIT base_config_dir:/clickhouse-config clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Setup Keeper Cluster name: project_name:roottests3cluster. Added instance name:s0_0_0 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env', '--project-name', 'roottests3cluster', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name: project_name:roottests3cluster. Added instance name:s0_0_1 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env', '--project-name', 'roottests3cluster', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name: project_name:roottests3cluster. Added instance name:s0_1_0 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env', '--project-name', 'roottests3cluster', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Starting cluster... Running tests in /ClickHouse/tests/integration/test_s3_cluster/test.py Cluster start called. is_up=False Docker networks for project roottests3cluster are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3cluster are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottests3cluster are DRIVER VOLUME NAME Cleanup called Docker networks for project roottests3cluster are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3cluster are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottests3cluster are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottests3cluster-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottests3cluster Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: s0_0_0 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/configs/config.d Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/database Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: s0_0_1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/database Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: s0_1_0 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/configs/config.d Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/database Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:6712d5cc610d', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/coordination', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/minio/certs/public.crt', 'RESOLVER_LOGS': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/resolver', 'RESOLVER_LOGS_FS': 'bind'} stored in /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --project-name roottests3cluster --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml pull] Stderr: s0_0_1 Skipped - Image is already being pulled by zoo1 Stderr: s0_1_0 Skipped - Image is already being pulled by zoo1 Stderr: proxy1 Skipped - Image is already being pulled by proxy2 Stderr: zoo2 Skipped - Image is already being pulled by zoo1 Stderr: zoo3 Skipped - Image is already being pulled by zoo1 Stderr: s0_0_0 Skipped - Image is already being pulled by zoo1 Stderr: zoo1 Pulling Stderr: minio1 Pulling Stderr: proxy2 Pulling Stderr: resolver Pulling Stderr: proxy2 Pulled Stderr: minio1 Pulled Stderr: resolver Pulled Stderr: zoo1 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/coordination', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/coordination', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/coordination'] Command:[docker compose --project-name roottests3cluster --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr:time="2025-04-02T00:13:08Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottests3cluster_default Creating Stderr: Network roottests3cluster_default Created Stderr: Container roottests3cluster-zoo2-1 Creating Stderr: Container roottests3cluster-zoo3-1 Creating Stderr: Container roottests3cluster-zoo1-1 Creating Stderr: Container roottests3cluster-zoo1-1 Created Stderr: Container roottests3cluster-zoo3-1 Created Stderr: Container roottests3cluster-zoo2-1 Created Stderr: Container roottests3cluster-zoo1-1 Starting Stderr: Container roottests3cluster-zoo3-1 Starting Stderr: Container roottests3cluster-zoo2-1 Starting Stderr: Container roottests3cluster-zoo1-1 Started Stderr: Container roottests3cluster-zoo2-1 Started Stderr: Container roottests3cluster-zoo3-1 Started Stderr:time="2025-04-02T00:13:09Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T00:13:09Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottests3cluster-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.3, port:2181, use_ssl:False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottests3cluster-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottests3cluster-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.2, port:2181, use_ssl:False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') Trying to create Minio instance by command docker compose --project-name roottests3cluster --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d Command:[docker compose --project-name roottests3cluster --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d] Stderr:time="2025-04-02T00:13:14Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Volume "roottests3cluster_data1-1" Creating Stderr: Volume "roottests3cluster_data1-1" Created Stderr:time="2025-04-02T00:13:14Z" level=warning msg="Found orphan containers ([roottests3cluster-zoo2-1 roottests3cluster-zoo1-1 roottests3cluster-zoo3-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up." Stderr: Container roottests3cluster-proxy1-1 Creating Stderr: Container roottests3cluster-proxy2-1 Creating Stderr: proxy2 The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested Stderr: proxy1 The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested Stderr: Container roottests3cluster-proxy1-1 Created Stderr: Container roottests3cluster-proxy2-1 Created Stderr: Container roottests3cluster-resolver-1 Creating Stderr: Container roottests3cluster-minio1-1 Creating Stderr: Container roottests3cluster-resolver-1 Created Stderr: Container roottests3cluster-minio1-1 Created Stderr: Container roottests3cluster-proxy2-1 Starting Stderr: Container roottests3cluster-proxy1-1 Starting Stderr: Container roottests3cluster-proxy2-1 Started Stderr: Container roottests3cluster-proxy1-1 Started Stderr: Container roottests3cluster-resolver-1 Starting Stderr: Container roottests3cluster-minio1-1 Starting Stderr: Container roottests3cluster-minio1-1 Started Stderr: Container roottests3cluster-resolver-1 Started Stderr:time="2025-04-02T00:13:15Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T00:13:15Z" level=debug msg="otel error" error="" Trying to connect to Minio... get_instance_ip instance_name=minio1 http://localhost:None "GET /v1.46/containers/roottests3cluster-minio1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=proxy1 http://localhost:None "GET /v1.46/containers/roottests3cluster-proxy1-1/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.1.8:9001 Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (2): 172.16.1.8:9001 Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (3): 172.16.1.8:9001 Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (4): 172.16.1.8:9001 Can't connect to Minio: HTTPConnectionPool(host='172.16.1.8', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) Starting new HTTP connection (5): 172.16.1.8:9001 http://172.16.1.8:9001 "GET / HTTP/1.1" 200 0 Connected to Minio. http://172.16.1.8:9001 "GET /root?location= HTTP/1.1" 404 0 http://172.16.1.8:9001 "PUT /root HTTP/1.1" 200 0 S3 bucket 'root' created http://172.16.1.8:9001 "GET /root2?location= HTTP/1.1" 404 0 http://172.16.1.8:9001 "PUT /root2 HTTP/1.1" 200 0 S3 bucket 'root2' created ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --project-name roottests3cluster --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --project-name roottests3cluster --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml up -d --no-recreate] Stderr: Container roottests3cluster-zoo3-1 Running Stderr: Container roottests3cluster-proxy2-1 Running Stderr: Container roottests3cluster-zoo2-1 Running Stderr: Container roottests3cluster-zoo1-1 Running Stderr: Container roottests3cluster-proxy1-1 Running Stderr: Container roottests3cluster-minio1-1 Running Stderr: Container roottests3cluster-s0_0_1-1 Creating Stderr: Container roottests3cluster-s0_0_0-1 Creating Stderr: Container roottests3cluster-resolver-1 Running Stderr: Container roottests3cluster-s0_1_0-1 Creating Stderr: Container roottests3cluster-s0_0_1-1 Created Stderr: Container roottests3cluster-s0_0_0-1 Created Stderr: Container roottests3cluster-s0_1_0-1 Created Stderr: Container roottests3cluster-s0_0_0-1 Starting Stderr: Container roottests3cluster-s0_1_0-1 Starting Stderr: Container roottests3cluster-s0_0_1-1 Starting Stderr: Container roottests3cluster-s0_1_0-1 Started Stderr: Container roottests3cluster-s0_0_0-1 Started Stderr: Container roottests3cluster-s0_0_1-1 Started ClickHouse instance created get_instance_ip instance_name=s0_0_0 http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_0-1/json HTTP/1.1" 200 None get_instance_ip instance_name=s0_0_0 http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_0-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in s0_0_0, ip: 172.16.1.10... http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_0-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None ClickHouse s0_0_0 started get_instance_ip instance_name=s0_0_1 http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=s0_0_1 http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in s0_0_1, ip: 172.16.1.11... http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/1b17d1a61b72d9ddcf781a06851ad48fc36b394984e7bf00b9359ad5ac45a9b0/json HTTP/1.1" 200 None ClickHouse s0_0_1 started get_instance_ip instance_name=s0_1_0 http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_1_0-1/json HTTP/1.1" 200 None get_instance_ip instance_name=s0_1_0 http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_1_0-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in s0_1_0, ip: 172.16.1.9... http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_1_0-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e83e0bfbab150fd687de8244f4231b4e60d6f5b0b0d7a7f0c0cef83c8b504f04/json HTTP/1.1" 200 None ClickHouse s0_1_0 started Cluster started http://172.16.1.8:9001 "PUT /root/data/clickhouse/part1.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/clickhouse/part123.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/database/part2.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/database/partition675.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_0.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_1.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_2.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_3.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_4.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_5.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_6.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_7.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_8.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_9.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_10.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_11.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_12.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_13.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_14.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_15.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_16.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_17.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_18.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_19.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_20.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_21.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_22.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_23.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_24.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_25.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_26.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_27.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_28.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_29.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_30.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_31.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_32.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_33.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_34.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_35.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_36.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_37.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_38.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_39.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_40.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_41.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_42.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_43.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_44.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_45.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_46.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_47.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_48.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_49.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_50.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_51.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_52.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_53.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_54.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_55.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_56.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_57.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_58.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_59.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_60.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_61.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_62.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_63.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_64.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_65.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_66.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_67.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_68.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_69.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_70.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_71.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_72.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_73.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_74.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_75.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_76.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_77.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_78.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_79.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_80.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_81.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_82.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_83.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_84.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_85.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_86.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_87.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_88.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_89.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_90.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_91.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_92.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_93.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_94.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_95.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_96.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_97.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_98.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "PUT /root/data/generated/file_99.csv HTTP/1.1" 200 0 http://172.16.1.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1" 200 0 Starting mock server s3_mock.py run container_id:roottests3cluster-resolver-1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlcXVlc3QsIHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKQHJvdXRlKCIvPF9idWNrZXQ+LzxfcGF0aDpwYXRoPiIpCmRlZiBzZXJ2ZXIoX2J1Y2tldCwgX3BhdGgpOgogICAgcmVzdWx0ID0gKAogICAgICAgIHJlcXVlc3QuaGVhZGVyc1siTXlDdXN0b21IZWFkZXIiXQogICAgICAgIGlmICJNeUN1c3RvbUhlYWRlciIgaW4gcmVxdWVzdC5oZWFkZXJzCiAgICAgICAgZWxzZSAidW5rbm93biIKICAgICkKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJ0ZXh0L3BsYWluIgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4ocmVzdWx0KSkKICAgIHJldHVybiByZXN1bHQKCgpAcm91dGUoIi8iKQpkZWYgcGluZygpOgogICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDIpCiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQo= | base64 --decode > s3_mock.py'] Command:[docker exec roottests3cluster-resolver-1 bash -c echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlcXVlc3QsIHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKQHJvdXRlKCIvPF9idWNrZXQ+LzxfcGF0aDpwYXRoPiIpCmRlZiBzZXJ2ZXIoX2J1Y2tldCwgX3BhdGgpOgogICAgcmVzdWx0ID0gKAogICAgICAgIHJlcXVlc3QuaGVhZGVyc1siTXlDdXN0b21IZWFkZXIiXQogICAgICAgIGlmICJNeUN1c3RvbUhlYWRlciIgaW4gcmVxdWVzdC5oZWFkZXJzCiAgICAgICAgZWxzZSAidW5rbm93biIKICAgICkKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJ0ZXh0L3BsYWluIgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4ocmVzdWx0KSkKICAgIHJldHVybiByZXN1bHQKCgpAcm91dGUoIi8iKQpkZWYgcGluZygpOgogICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDIpCiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQo= | base64 --decode > s3_mock.py] run container_id:roottests3cluster-resolver-1 detach:True nothrow:False cmd: ['bash', '-c', 'python3 s3_mock.py 8080 >/var/log/resolver/s3_mock.log 2>/var/log/resolver/s3_mock.err.log'] Command:[docker exec roottests3cluster-resolver-1 bash -c python3 s3_mock.py 8080 >/var/log/resolver/s3_mock.log 2>/var/log/resolver/s3_mock.err.log] run container_id:roottests3cluster-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] Command:[docker exec roottests3cluster-resolver-1 curl -s http://localhost:8080/] Exitcode:7 run container_id:roottests3cluster-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] Command:[docker exec roottests3cluster-resolver-1 curl -s http://localhost:8080/] Stdout:OK s3_mock.py answered OK on attempt 2 Mock server s3_mock.py started ------------------------------ Captured log setup ------------------------------ 2025-04-02 00:12:57 [ 630 ] DEBUG : Command:[docker ps | wc -l] (cluster.py:122, run_and_check) 2025-04-02 00:12:57 [ 630 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check) 2025-04-02 00:12:57 [ 630 ] DEBUG : No running containers (conftest.py:96, cleanup_environment) 2025-04-02 00:12:57 [ 630 ] DEBUG : Pruning Docker networks (conftest.py:98, cleanup_environment) 2025-04-02 00:12:57 [ 630 ] DEBUG : Command:[docker network prune --force] (cluster.py:122, run_and_check) 2025-04-02 00:12:57 [ 630 ] DEBUG : Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] (cluster.py:122, run_and_check) 2025-04-02 00:12:57 [ 630 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:146, run_and_check) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV HOSTNAME 34bca11fe2c4 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV SHLVL 0 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV HOME /root (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV OLDPWD / (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_HELPER_TAG 5dc43a6382f0 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV PYTHONUNBUFFERED 1 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV UBSAN_OPTIONS print_stacktrace=1 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV PYTEST_ADDOPTS -rfEps --run-id=2 --color=no --durations=0 test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference -vvv (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV COMPOSE_HTTP_TIMEOUT 600 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_KERBERIZED_HADOOP_TAG latest (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_CHANNEL stable (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_CLIENT_TIMEOUT 300 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV PWD /ClickHouse/tests/integration (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV TZ Etc/UTC (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV DOCKER_BASE_TAG 6712d5cc610d (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV LC_CTYPE C.UTF-8 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV INTEGRATION_TESTS_RUN_ID 2 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV WORKER_FREE_PORTS 30000 30001 30002 30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014 30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026 30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038 30039 30040 30041 30042 30043 30044 30045 30046 30047 30048 30049 (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : ENV PYTEST_CURRENT_TEST test_s3_cluster/test.py::test_distributed_insert_select_with_replicated (setup) (cluster.py:450, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:774, __init__) 2025-04-02 00:12:57 [ 630 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1729, add_instance) 2025-04-02 00:12:57 [ 630 ] DEBUG : Setup Keeper (cluster.py:1069, setup_keeper_cmd) 2025-04-02 00:12:57 [ 630 ] DEBUG : Cluster name: project_name:roottests3cluster. Added instance name:s0_0_0 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env', '--project-name', 'roottests3cluster', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2025, add_instance) 2025-04-02 00:12:57 [ 630 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1729, add_instance) 2025-04-02 00:12:57 [ 630 ] DEBUG : Cluster name: project_name:roottests3cluster. Added instance name:s0_0_1 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env', '--project-name', 'roottests3cluster', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2025, add_instance) 2025-04-02 00:12:57 [ 630 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1729, add_instance) 2025-04-02 00:12:57 [ 630 ] DEBUG : Cluster name: project_name:roottests3cluster. Added instance name:s0_1_0 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env', '--project-name', 'roottests3cluster', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2025, add_instance) 2025-04-02 00:12:57 [ 630 ] INFO : Starting cluster... (test.py:94, started_cluster) 2025-04-02 00:12:57 [ 630 ] INFO : Running tests in /ClickHouse/tests/integration/test_s3_cluster/test.py (cluster.py:2793, start) 2025-04-02 00:12:57 [ 630 ] DEBUG : Cluster start called. is_up=False (cluster.py:2800, start) 2025-04-02 00:12:57 [ 630 ] DEBUG : Docker networks for project roottests3cluster are NETWORK ID NAME DRIVER SCOPE (cluster.py:873, print_all_docker_pieces) 2025-04-02 00:12:57 [ 630 ] DEBUG : Docker containers for project roottests3cluster are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:881, print_all_docker_pieces) 2025-04-02 00:12:57 [ 630 ] DEBUG : Docker volumes for project roottests3cluster are DRIVER VOLUME NAME (cluster.py:889, print_all_docker_pieces) 2025-04-02 00:12:57 [ 630 ] DEBUG : Cleanup called (cluster.py:894, cleanup) 2025-04-02 00:12:57 [ 630 ] DEBUG : Docker networks for project roottests3cluster are NETWORK ID NAME DRIVER SCOPE (cluster.py:873, print_all_docker_pieces) 2025-04-02 00:12:57 [ 630 ] DEBUG : Docker containers for project roottests3cluster are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:881, print_all_docker_pieces) 2025-04-02 00:12:57 [ 630 ] DEBUG : Docker volumes for project roottests3cluster are DRIVER VOLUME NAME (cluster.py:889, print_all_docker_pieces) 2025-04-02 00:12:57 [ 630 ] DEBUG : Command:[docker container list --all --filter name='^/roottests3cluster-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:122, run_and_check) 2025-04-02 00:12:57 [ 630 ] DEBUG : Unstopped containers: {} (cluster.py:908, cleanup) 2025-04-02 00:12:57 [ 630 ] DEBUG : No running containers for project: roottests3cluster (cluster.py:922, cleanup) 2025-04-02 00:12:57 [ 630 ] DEBUG : Trying to prune unused networks... (cluster.py:928, cleanup) 2025-04-02 00:12:57 [ 630 ] DEBUG : Trying to prune unused images... (cluster.py:944, cleanup) 2025-04-02 00:12:57 [ 630 ] DEBUG : Command:[docker image prune -f] (cluster.py:122, run_and_check) 2025-04-02 00:12:57 [ 630 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:146, run_and_check) 2025-04-02 00:12:57 [ 630 ] DEBUG : Images pruned (cluster.py:947, cleanup) 2025-04-02 00:12:57 [ 630 ] DEBUG : Trying to prune unused volumes... (cluster.py:953, cleanup) 2025-04-02 00:12:57 [ 630 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:122, run_and_check) 2025-04-02 00:12:57 [ 630 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check) 2025-04-02 00:12:57 [ 630 ] DEBUG : Volumes pruned: 1 (cluster.py:958, cleanup) 2025-04-02 00:12:57 [ 630 ] DEBUG : Setup directory for instance: s0_0_0 (cluster.py:2813, start) 2025-04-02 00:12:57 [ 630 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4639, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Create directory for common tests configuration (cluster.py:4644, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Copy common configuration from helpers (cluster.py:4664, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Generate and write macros file (cluster.py:4716, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/configs/config.d (cluster.py:4752, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/database (cluster.py:4769, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/logs (cluster.py:4780, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4864, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Setup directory for instance: s0_0_1 (cluster.py:2813, start) 2025-04-02 00:12:57 [ 630 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4639, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Create directory for common tests configuration (cluster.py:4644, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Copy common configuration from helpers (cluster.py:4664, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Generate and write macros file (cluster.py:4716, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/configs/config.d (cluster.py:4752, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/database (cluster.py:4769, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/logs (cluster.py:4780, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4864, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Setup directory for instance: s0_1_0 (cluster.py:2813, start) 2025-04-02 00:12:57 [ 630 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4639, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Create directory for common tests configuration (cluster.py:4644, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Copy common configuration from helpers (cluster.py:4664, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Generate and write macros file (cluster.py:4716, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/configs/config.d (cluster.py:4752, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/database (cluster.py:4769, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/logs (cluster.py:4780, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4864, create_dir) 2025-04-02 00:12:57 [ 630 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:6712d5cc610d', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/coordination', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/minio/certs/public.crt', 'RESOLVER_LOGS': '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/resolver', 'RESOLVER_LOGS_FS': 'bind'} stored in /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env (cluster.py:97, _create_env_file) 2025-04-02 00:12:57 [ 630 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-04-02 00:12:57 [ 630 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-04-02 00:12:57 [ 630 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-04-02 00:12:57 [ 630 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-04-02 00:12:57 [ 630 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2025-04-02 00:12:57 [ 630 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --project-name roottests3cluster --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml pull] (cluster.py:122, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: s0_0_1 Skipped - Image is already being pulled by zoo1 (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: s0_1_0 Skipped - Image is already being pulled by zoo1 (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: proxy1 Skipped - Image is already being pulled by proxy2 (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: zoo2 Skipped - Image is already being pulled by zoo1 (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: zoo3 Skipped - Image is already being pulled by zoo1 (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: s0_0_0 Skipped - Image is already being pulled by zoo1 (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: zoo1 Pulling (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: minio1 Pulling (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: proxy2 Pulling (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: resolver Pulling (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: proxy2 Pulled (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: minio1 Pulled (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: resolver Pulled (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Stderr: zoo1 Pulled (cluster.py:148, run_and_check) 2025-04-02 00:13:08 [ 630 ] DEBUG : Setup ZooKeeper (cluster.py:2854, start) 2025-04-02 00:13:08 [ 630 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper1/coordination', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper2/coordination', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-2/keeper3/coordination'] (cluster.py:2855, start) 2025-04-02 00:13:08 [ 630 ] DEBUG : Command:[docker compose --project-name roottests3cluster --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] (cluster.py:122, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr:time="2025-04-02T00:13:08Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Network roottests3cluster_default Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Network roottests3cluster_default Created (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Created (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Created (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Created (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Starting (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Starting (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Starting (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Started (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Started (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Started (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr:time="2025-04-02T00:13:09Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Stderr:time="2025-04-02T00:13:09Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-02 00:13:09 [ 630 ] DEBUG : Wait ZooKeeper to start (cluster.py:2466, wait_zookeeper_to_start) 2025-04-02 00:13:09 [ 630 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2082, get_instance_ip) 2025-04-02 00:13:09 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-zoo1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:09 [ 630 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.1.3, port:2181, use_ssl:False (cluster.py:3341, get_kazoo_client) 2025-04-02 00:13:09 [ 630 ] INFO : Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False (connection.py:650, _connect) 2025-04-02 00:13:09 [ 630 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-04-02 00:13:09 [ 630 ] INFO : Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False (connection.py:650, _connect) 2025-04-02 00:13:09 [ 630 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-04-02 00:13:09 [ 630 ] INFO : Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False (connection.py:650, _connect) 2025-04-02 00:13:09 [ 630 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-04-02 00:13:10 [ 630 ] INFO : Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False (connection.py:650, _connect) 2025-04-02 00:13:10 [ 630 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-04-02 00:13:10 [ 630 ] INFO : Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False (connection.py:650, _connect) 2025-04-02 00:13:10 [ 630 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-04-02 00:13:11 [ 630 ] INFO : Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False (connection.py:650, _connect) 2025-04-02 00:13:11 [ 630 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-04-02 00:13:13 [ 630 ] INFO : Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False (connection.py:650, _connect) 2025-04-02 00:13:13 [ 630 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-04-02 00:13:13 [ 630 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-04-02 00:13:13 [ 630 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2025-04-02 00:13:13 [ 630 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2025-04-02 00:13:13 [ 630 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2025-04-02 00:13:13 [ 630 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-04-02 00:13:13 [ 630 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-04-02 00:13:13 [ 630 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-04-02 00:13:14 [ 630 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-04-02 00:13:14 [ 630 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-04-02 00:13:14 [ 630 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:2082, get_instance_ip) 2025-04-02 00:13:14 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-zoo2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:14 [ 630 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.1.4, port:2181, use_ssl:False (cluster.py:3341, get_kazoo_client) 2025-04-02 00:13:14 [ 630 ] INFO : Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False (connection.py:650, _connect) 2025-04-02 00:13:14 [ 630 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-04-02 00:13:14 [ 630 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-04-02 00:13:14 [ 630 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2025-04-02 00:13:14 [ 630 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2025-04-02 00:13:14 [ 630 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2025-04-02 00:13:14 [ 630 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-04-02 00:13:14 [ 630 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-04-02 00:13:14 [ 630 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-04-02 00:13:14 [ 630 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-04-02 00:13:14 [ 630 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-04-02 00:13:14 [ 630 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:2082, get_instance_ip) 2025-04-02 00:13:14 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-zoo3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:14 [ 630 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.1.2, port:2181, use_ssl:False (cluster.py:3341, get_kazoo_client) 2025-04-02 00:13:14 [ 630 ] INFO : Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False (connection.py:650, _connect) 2025-04-02 00:13:14 [ 630 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-04-02 00:13:14 [ 630 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-04-02 00:13:14 [ 630 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2025-04-02 00:13:14 [ 630 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2025-04-02 00:13:14 [ 630 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2025-04-02 00:13:14 [ 630 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-04-02 00:13:14 [ 630 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-04-02 00:13:14 [ 630 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-04-02 00:13:14 [ 630 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-04-02 00:13:14 [ 630 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-04-02 00:13:14 [ 630 ] DEBUG : All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') (cluster.py:2482, wait_zookeeper_nodes_to_start) 2025-04-02 00:13:14 [ 630 ] INFO : Trying to create Minio instance by command docker compose --project-name roottests3cluster --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d (cluster.py:3132, start) 2025-04-02 00:13:14 [ 630 ] DEBUG : Command:[docker compose --project-name roottests3cluster --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d] (cluster.py:122, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr:time="2025-04-02T00:13:14Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Volume "roottests3cluster_data1-1" Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Volume "roottests3cluster_data1-1" Created (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr:time="2025-04-02T00:13:14Z" level=warning msg="Found orphan containers ([roottests3cluster-zoo2-1 roottests3cluster-zoo1-1 roottests3cluster-zoo3-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up." (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: proxy2 The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: proxy1 The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Created (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Created (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Created (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Created (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Starting (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Starting (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Started (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Started (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Starting (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Starting (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Started (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Started (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr:time="2025-04-02T00:13:15Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] DEBUG : Stderr:time="2025-04-02T00:13:15Z" level=debug msg="otel error" error="" (cluster.py:148, run_and_check) 2025-04-02 00:13:15 [ 630 ] INFO : Trying to connect to Minio... (cluster.py:3138, start) 2025-04-02 00:13:15 [ 630 ] DEBUG : get_instance_ip instance_name=minio1 (cluster.py:2082, get_instance_ip) 2025-04-02 00:13:15 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-minio1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:15 [ 630 ] DEBUG : get_instance_ip instance_name=proxy1 (cluster.py:2082, get_instance_ip) 2025-04-02 00:13:15 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-proxy1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:15 [ 630 ] DEBUG : Starting new HTTP connection (1): 172.16.1.8:9001 (connectionpool.py:245, _new_conn) 2025-04-02 00:13:15 [ 630 ] DEBUG : Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2025-04-02 00:13:15 [ 630 ] WARNING : Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2025-04-02 00:13:15 [ 630 ] DEBUG : Starting new HTTP connection (2): 172.16.1.8:9001 (connectionpool.py:245, _new_conn) 2025-04-02 00:13:15 [ 630 ] DEBUG : Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2025-04-02 00:13:15 [ 630 ] WARNING : Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2025-04-02 00:13:15 [ 630 ] DEBUG : Starting new HTTP connection (3): 172.16.1.8:9001 (connectionpool.py:245, _new_conn) 2025-04-02 00:13:15 [ 630 ] DEBUG : Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2025-04-02 00:13:15 [ 630 ] WARNING : Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2025-04-02 00:13:15 [ 630 ] DEBUG : Starting new HTTP connection (4): 172.16.1.8:9001 (connectionpool.py:245, _new_conn) 2025-04-02 00:13:15 [ 630 ] DEBUG : Can't connect to Minio: HTTPConnectionPool(host='172.16.1.8', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2637, wait_minio_to_start) 2025-04-02 00:13:16 [ 630 ] DEBUG : Starting new HTTP connection (5): 172.16.1.8:9001 (connectionpool.py:245, _new_conn) 2025-04-02 00:13:16 [ 630 ] DEBUG : http://172.16.1.8:9001 "GET / HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:16 [ 630 ] DEBUG : Connected to Minio. (cluster.py:2617, wait_minio_to_start) 2025-04-02 00:13:16 [ 630 ] DEBUG : http://172.16.1.8:9001 "GET /root?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:16 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:16 [ 630 ] DEBUG : S3 bucket 'root' created (cluster.py:2632, wait_minio_to_start) 2025-04-02 00:13:16 [ 630 ] DEBUG : http://172.16.1.8:9001 "GET /root2?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:16 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root2 HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:16 [ 630 ] DEBUG : S3 bucket 'root2' created (cluster.py:2632, wait_minio_to_start) 2025-04-02 00:13:16 [ 630 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --project-name roottests3cluster --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml up -d --no-recreate') (cluster.py:3200, start) 2025-04-02 00:13:16 [ 630 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --project-name roottests3cluster --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml up -d --no-recreate] (cluster.py:122, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Running (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Running (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Running (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Running (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Running (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Running (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_1-1 Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_0-1 Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Running (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_1_0-1 Creating (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_1-1 Created (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_0-1 Created (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_1_0-1 Created (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_0-1 Starting (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_1_0-1 Starting (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_1-1 Starting (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_1_0-1 Started (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_0-1 Started (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_1-1 Started (cluster.py:148, run_and_check) 2025-04-02 00:13:17 [ 630 ] DEBUG : ClickHouse instance created (cluster.py:3208, start) 2025-04-02 00:13:17 [ 630 ] DEBUG : get_instance_ip instance_name=s0_0_0 (cluster.py:2082, get_instance_ip) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : get_instance_ip instance_name=s0_0_0 (cluster.py:2092, get_instance_global_ipv6) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : Waiting for ClickHouse start in s0_0_0, ip: 172.16.1.10... (cluster.py:3216, start) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/cfd504cf4b79eb7df643b173e2e0337057a9f28638dd2386e86520248be00644/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : ClickHouse s0_0_0 started (cluster.py:3220, start) 2025-04-02 00:13:17 [ 630 ] DEBUG : get_instance_ip instance_name=s0_0_1 (cluster.py:2082, get_instance_ip) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : get_instance_ip instance_name=s0_0_1 (cluster.py:2092, get_instance_global_ipv6) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : Waiting for ClickHouse start in s0_0_1, ip: 172.16.1.11... (cluster.py:3216, start) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_0_1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/1b17d1a61b72d9ddcf781a06851ad48fc36b394984e7bf00b9359ad5ac45a9b0/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : ClickHouse s0_0_1 started (cluster.py:3220, start) 2025-04-02 00:13:17 [ 630 ] DEBUG : get_instance_ip instance_name=s0_1_0 (cluster.py:2082, get_instance_ip) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_1_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : get_instance_ip instance_name=s0_1_0 (cluster.py:2092, get_instance_global_ipv6) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_1_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : Waiting for ClickHouse start in s0_1_0, ip: 172.16.1.9... (cluster.py:3216, start) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-s0_1_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://localhost:None "GET /v1.46/containers/e83e0bfbab150fd687de8244f4231b4e60d6f5b0b0d7a7f0c0cef83c8b504f04/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : ClickHouse s0_1_0 started (cluster.py:3220, start) 2025-04-02 00:13:17 [ 630 ] INFO : Cluster started (test.py:96, started_cluster) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/clickhouse/part1.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/clickhouse/part123.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/database/part2.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/database/partition675.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_0.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_1.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_2.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_3.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_4.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_5.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_6.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_7.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_8.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_9.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_10.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_11.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_12.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_13.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_14.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_15.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_16.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_17.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_18.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_19.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_20.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_21.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_22.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_23.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_24.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_25.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_26.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_27.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_28.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_29.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_30.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_31.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_32.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_33.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_34.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_35.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_36.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_37.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_38.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_39.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_40.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_41.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_42.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_43.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_44.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_45.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_46.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_47.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_48.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_49.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_50.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_51.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_52.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_53.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_54.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_55.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_56.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_57.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_58.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_59.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_60.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_61.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_62.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_63.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_64.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_65.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_66.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_67.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_68.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_69.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_70.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_71.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_72.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_73.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_74.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_75.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_76.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_77.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_78.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_79.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_80.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:17 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_81.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_82.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_83.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_84.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_85.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_86.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_87.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_88.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_89.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_90.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_91.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_92.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_93.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_94.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_95.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_96.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_97.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_98.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "PUT /root/data/generated/file_99.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] DEBUG : http://172.16.1.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-04-02 00:13:18 [ 630 ] INFO : Starting mock server s3_mock.py (mock_servers.py:18, start_mock_servers) 2025-04-02 00:13:18 [ 630 ] DEBUG : run container_id:roottests3cluster-resolver-1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlcXVlc3QsIHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKQHJvdXRlKCIvPF9idWNrZXQ+LzxfcGF0aDpwYXRoPiIpCmRlZiBzZXJ2ZXIoX2J1Y2tldCwgX3BhdGgpOgogICAgcmVzdWx0ID0gKAogICAgICAgIHJlcXVlc3QuaGVhZGVyc1siTXlDdXN0b21IZWFkZXIiXQogICAgICAgIGlmICJNeUN1c3RvbUhlYWRlciIgaW4gcmVxdWVzdC5oZWFkZXJzCiAgICAgICAgZWxzZSAidW5rbm93biIKICAgICkKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJ0ZXh0L3BsYWluIgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4ocmVzdWx0KSkKICAgIHJldHVybiByZXN1bHQKCgpAcm91dGUoIi8iKQpkZWYgcGluZygpOgogICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDIpCiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQo= | base64 --decode > s3_mock.py'] (cluster.py:2126, exec_in_container) 2025-04-02 00:13:18 [ 630 ] DEBUG : Command:[docker exec roottests3cluster-resolver-1 bash -c echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlcXVlc3QsIHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKQHJvdXRlKCIvPF9idWNrZXQ+LzxfcGF0aDpwYXRoPiIpCmRlZiBzZXJ2ZXIoX2J1Y2tldCwgX3BhdGgpOgogICAgcmVzdWx0ID0gKAogICAgICAgIHJlcXVlc3QuaGVhZGVyc1siTXlDdXN0b21IZWFkZXIiXQogICAgICAgIGlmICJNeUN1c3RvbUhlYWRlciIgaW4gcmVxdWVzdC5oZWFkZXJzCiAgICAgICAgZWxzZSAidW5rbm93biIKICAgICkKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJ0ZXh0L3BsYWluIgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4ocmVzdWx0KSkKICAgIHJldHVybiByZXN1bHQKCgpAcm91dGUoIi8iKQpkZWYgcGluZygpOgogICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDIpCiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQo= | base64 --decode > s3_mock.py] (cluster.py:122, run_and_check) 2025-04-02 00:13:18 [ 630 ] DEBUG : run container_id:roottests3cluster-resolver-1 detach:True nothrow:False cmd: ['bash', '-c', 'python3 s3_mock.py 8080 >/var/log/resolver/s3_mock.log 2>/var/log/resolver/s3_mock.err.log'] (cluster.py:2126, exec_in_container) 2025-04-02 00:13:18 [ 630 ] DEBUG : Command:[docker exec roottests3cluster-resolver-1 bash -c python3 s3_mock.py 8080 >/var/log/resolver/s3_mock.log 2>/var/log/resolver/s3_mock.err.log] (cluster.py:122, run_and_check) 2025-04-02 00:13:18 [ 630 ] DEBUG : run container_id:roottests3cluster-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2126, exec_in_container) 2025-04-02 00:13:18 [ 630 ] DEBUG : Command:[docker exec roottests3cluster-resolver-1 curl -s http://localhost:8080/] (cluster.py:122, run_and_check) 2025-04-02 00:13:18 [ 630 ] DEBUG : Exitcode:7 (cluster.py:150, run_and_check) 2025-04-02 00:13:19 [ 630 ] DEBUG : run container_id:roottests3cluster-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2126, exec_in_container) 2025-04-02 00:13:19 [ 630 ] DEBUG : Command:[docker exec roottests3cluster-resolver-1 curl -s http://localhost:8080/] (cluster.py:122, run_and_check) 2025-04-02 00:13:19 [ 630 ] DEBUG : Stdout:OK (cluster.py:146, run_and_check) 2025-04-02 00:13:19 [ 630 ] DEBUG : s3_mock.py answered OK on attempt 2 (mock_servers.py:67, start_mock_servers) 2025-04-02 00:13:19 [ 630 ] INFO : Mock server s3_mock.py started (mock_servers.py:82, start_mock_servers) ----------------------------- Captured stderr call ----------------------------- Executing query DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC; on s0_0_0 Executing query CREATE TABLE insert_select_replicated_local ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/insert_select_with_replicated', '{replica}') ORDER BY (a, b); on s0_0_0 Executing query SYSTEM STOP FETCHES; on s0_0_0 Executing query SYSTEM STOP MERGES; on s0_0_0 Executing query SYSTEM STOP FETCHES; on s0_0_1 Executing query SYSTEM STOP MERGES; on s0_0_1 Executing query INSERT INTO insert_select_replicated_local SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV','a String, b UInt64' ) SETTINGS parallel_distributed_insert_select=1; on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-02 00:13:19 [ 630 ] DEBUG : Executing query DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC; on s0_0_0 (cluster.py:3677, query) 2025-04-02 00:13:19 [ 630 ] DEBUG : Executing query CREATE TABLE insert_select_replicated_local ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/insert_select_with_replicated', '{replica}') ORDER BY (a, b); on s0_0_0 (cluster.py:3677, query) 2025-04-02 00:13:19 [ 630 ] DEBUG : Executing query SYSTEM STOP FETCHES; on s0_0_0 (cluster.py:3677, query) 2025-04-02 00:13:19 [ 630 ] DEBUG : Executing query SYSTEM STOP MERGES; on s0_0_0 (cluster.py:3677, query) 2025-04-02 00:13:19 [ 630 ] DEBUG : Executing query SYSTEM STOP FETCHES; on s0_0_1 (cluster.py:3677, query) 2025-04-02 00:13:19 [ 630 ] DEBUG : Executing query SYSTEM STOP MERGES; on s0_0_1 (cluster.py:3677, query) 2025-04-02 00:13:20 [ 630 ] DEBUG : Executing query INSERT INTO insert_select_replicated_local SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV','a String, b UInt64' ) SETTINGS parallel_distributed_insert_select=1; on s0_0_0 (cluster.py:3677, query) ________ test_parallel_distributed_insert_select_with_schema_inference _________ started_cluster = def test_parallel_distributed_insert_select_with_schema_inference(started_cluster): node = started_cluster.instances["s0_0_0"] node.query( """DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC;""" ) node.query( """ CREATE TABLE parallel_insert_select ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/parallel_insert_select', '{replica}') ORDER BY (a, b); """ ) > node.query( """ INSERT INTO parallel_insert_select SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV' ) SETTINGS parallel_distributed_insert_select=1, use_structure_from_insertion_table_in_table_functions=0; """ ) test_s3_cluster/test.py:448: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 49, stderr: Received exception from server (version 24.12.2): E Code: 49. DB::Exception: Received from 172.16.1.10:9000. DB::Exception: Replica info is not initialized: While executing Remote. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d78d79b E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000867958c E 2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000008686b0b E 3. DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x00000000112e0637 E 4. DB::RemoteQueryExecutor::readAsync() @ 0x00000000112e1b76 E 5. DB::RemoteSource::tryGenerate() @ 0x0000000013aefa30 E 6. DB::ISource::work() @ 0x0000000013787127 E 7. DB::ExecutionThreadContext::executeTask() @ 0x00000000137a1947 E 8. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x0000000013794c25 E 9. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000013793fb6 E 10. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000137930f7 E 11. ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x000000000d86a102 E 12. void* std::__thread_proxy[abi:v15007]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000000d8714ba E 13. ? @ 0x00007fea98874ac3 E 14. ? @ 0x00007fea98906850 E . (LOGICAL_ERROR) E (query: INSERT INTO parallel_insert_select SELECT * FROM s3Cluster( E 'first_shard', E 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV' E ) SETTINGS parallel_distributed_insert_select=1, use_structure_from_insertion_table_in_table_functions=0;) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC; on s0_0_0 Executing query CREATE TABLE parallel_insert_select ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/parallel_insert_select', '{replica}') ORDER BY (a, b); on s0_0_0 Executing query INSERT INTO parallel_insert_select SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV' ) SETTINGS parallel_distributed_insert_select=1, use_structure_from_insertion_table_in_table_functions=0; on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-02 00:13:20 [ 630 ] DEBUG : Executing query DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC; on s0_0_0 (cluster.py:3677, query) 2025-04-02 00:13:20 [ 630 ] DEBUG : Executing query CREATE TABLE parallel_insert_select ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/parallel_insert_select', '{replica}') ORDER BY (a, b); on s0_0_0 (cluster.py:3677, query) 2025-04-02 00:13:20 [ 630 ] DEBUG : Executing query INSERT INTO parallel_insert_select SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV' ) SETTINGS parallel_distributed_insert_select=1, use_structure_from_insertion_table_in_table_functions=0; on s0_0_0 (cluster.py:3677, query) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --project-name roottests3cluster --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml stop --timeout 20] Stderr: Container roottests3cluster-s0_1_0-1 Stopping Stderr: Container roottests3cluster-s0_0_0-1 Stopping Stderr: Container roottests3cluster-resolver-1 Stopping Stderr: Container roottests3cluster-s0_0_1-1 Stopping Stderr: Container roottests3cluster-s0_1_0-1 Stopped Stderr: Container roottests3cluster-s0_0_0-1 Stopped Stderr: Container roottests3cluster-minio1-1 Stopping Stderr: Container roottests3cluster-s0_0_1-1 Stopped Stderr: Container roottests3cluster-zoo3-1 Stopping Stderr: Container roottests3cluster-zoo1-1 Stopping Stderr: Container roottests3cluster-zoo2-1 Stopping Stderr: Container roottests3cluster-zoo3-1 Stopped Stderr: Container roottests3cluster-zoo2-1 Stopped Stderr: Container roottests3cluster-zoo1-1 Stopped Stderr: Container roottests3cluster-minio1-1 Stopped Stderr: Container roottests3cluster-resolver-1 Stopped Stderr: Container roottests3cluster-proxy1-1 Stopping Stderr: Container roottests3cluster-proxy2-1 Stopping Stderr: Container roottests3cluster-proxy1-1 Stopped Stderr: Container roottests3cluster-proxy2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --project-name roottests3cluster --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml down --volumes] Stderr: Container roottests3cluster-resolver-1 Stopping Stderr: Container roottests3cluster-s0_0_1-1 Stopping Stderr: Container roottests3cluster-s0_1_0-1 Stopping Stderr: Container roottests3cluster-s0_0_0-1 Stopping Stderr: Container roottests3cluster-resolver-1 Stopped Stderr: Container roottests3cluster-resolver-1 Removing Stderr: Container roottests3cluster-s0_0_1-1 Stopped Stderr: Container roottests3cluster-s0_0_1-1 Removing Stderr: Container roottests3cluster-s0_0_0-1 Stopped Stderr: Container roottests3cluster-s0_0_0-1 Removing Stderr: Container roottests3cluster-s0_1_0-1 Stopped Stderr: Container roottests3cluster-s0_1_0-1 Removing Stderr: Container roottests3cluster-s0_0_0-1 Removed Stderr: Container roottests3cluster-minio1-1 Stopping Stderr: Container roottests3cluster-s0_0_1-1 Removed Stderr: Container roottests3cluster-s0_1_0-1 Removed Stderr: Container roottests3cluster-zoo1-1 Stopping Stderr: Container roottests3cluster-zoo2-1 Stopping Stderr: Container roottests3cluster-resolver-1 Removed Stderr: Container roottests3cluster-zoo3-1 Stopping Stderr: Container roottests3cluster-minio1-1 Stopped Stderr: Container roottests3cluster-minio1-1 Removing Stderr: Container roottests3cluster-zoo1-1 Stopped Stderr: Container roottests3cluster-zoo1-1 Removing Stderr: Container roottests3cluster-zoo2-1 Stopped Stderr: Container roottests3cluster-zoo2-1 Removing Stderr: Container roottests3cluster-zoo3-1 Stopped Stderr: Container roottests3cluster-zoo3-1 Removing Stderr: Container roottests3cluster-zoo1-1 Removed Stderr: Container roottests3cluster-zoo2-1 Removed Stderr: Container roottests3cluster-zoo3-1 Removed Stderr: Container roottests3cluster-minio1-1 Removed Stderr: Container roottests3cluster-proxy2-1 Stopping Stderr: Container roottests3cluster-proxy1-1 Stopping Stderr: Container roottests3cluster-proxy1-1 Stopped Stderr: Container roottests3cluster-proxy1-1 Removing Stderr: Container roottests3cluster-proxy2-1 Stopped Stderr: Container roottests3cluster-proxy2-1 Removing Stderr: Container roottests3cluster-proxy1-1 Removed Stderr: Container roottests3cluster-proxy2-1 Removed Stderr: Volume roottests3cluster_data1-1 Removing Stderr: Network roottests3cluster_default Removing Stderr: Volume roottests3cluster_data1-1 Removed Stderr: Network roottests3cluster_default Removed Cleanup called Docker networks for project roottests3cluster are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3cluster are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottests3cluster are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottests3cluster-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottests3cluster Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 ---------------------------- Captured log teardown ----------------------------- 2025-04-02 00:13:21 [ 630 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --project-name roottests3cluster --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml stop --timeout 20] (cluster.py:122, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_1_0-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_0-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_1-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_1_0-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_0-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_1-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-02 00:13:41 [ 630 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/.env --project-name roottests3cluster --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-2/s0_1_0/docker-compose.yml down --volumes] (cluster.py:122, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_1-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_1_0-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_0-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_1-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_1-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_0-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_0-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_1_0-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_1_0-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_0-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_0_1-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-s0_1_0-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-resolver-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo1-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo2-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-zoo3-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-minio1-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Stopping (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Stopped (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy1-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Container roottests3cluster-proxy2-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Volume roottests3cluster_data1-1 Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Network roottests3cluster_default Removing (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Volume roottests3cluster_data1-1 Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stderr: Network roottests3cluster_default Removed (cluster.py:148, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Cleanup called (cluster.py:894, cleanup) 2025-04-02 00:13:42 [ 630 ] DEBUG : Docker networks for project roottests3cluster are NETWORK ID NAME DRIVER SCOPE (cluster.py:873, print_all_docker_pieces) 2025-04-02 00:13:42 [ 630 ] DEBUG : Docker containers for project roottests3cluster are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:881, print_all_docker_pieces) 2025-04-02 00:13:42 [ 630 ] DEBUG : Docker volumes for project roottests3cluster are DRIVER VOLUME NAME (cluster.py:889, print_all_docker_pieces) 2025-04-02 00:13:42 [ 630 ] DEBUG : Command:[docker container list --all --filter name='^/roottests3cluster-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:122, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Unstopped containers: {} (cluster.py:908, cleanup) 2025-04-02 00:13:42 [ 630 ] DEBUG : No running containers for project: roottests3cluster (cluster.py:922, cleanup) 2025-04-02 00:13:42 [ 630 ] DEBUG : Trying to prune unused networks... (cluster.py:928, cleanup) 2025-04-02 00:13:42 [ 630 ] DEBUG : Trying to prune unused images... (cluster.py:944, cleanup) 2025-04-02 00:13:42 [ 630 ] DEBUG : Command:[docker image prune -f] (cluster.py:122, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:146, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Images pruned (cluster.py:947, cleanup) 2025-04-02 00:13:42 [ 630 ] DEBUG : Trying to prune unused volumes... (cluster.py:953, cleanup) 2025-04-02 00:13:42 [ 630 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:122, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check) 2025-04-02 00:13:42 [ 630 ] DEBUG : Volumes pruned: 1 (cluster.py:958, cleanup) ============================== slowest durations =============================== 22.14s setup test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 21.81s teardown test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 0.81s call test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 0.45s call test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 0.00s teardown test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 0.00s setup test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference =========================== short test summary info ============================ FAILED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated FAILED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference ============================== 2 failed in 45.54s ============================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 528, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_bk5u9t --privileged --dns-search='.' --memory=30709035008 --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=6712d5cc610d -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS=" -rfEps --run-id=2 --color=no --durations=0 test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference -vvv" altinityinfra/integration-tests-runner:cd6390247eca ' returned non-zero exit status 1.